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ABSTRACT 

A central question to evaluative research at all 
levels isr Who should do it? In evaluating drug abuse programs three 
groups might be involved: treatment personnel^ administrative or 
research personnel r and outside research professionals* This paper 
presents some advantages and disadvantages to the involvement of each 
of these groups within the context of five core dimensions: design 
biasy response biasr trust and access^ expertise^ and experimental 
control. At the Veterans Administration Hospital in Palo AltOr 
California, a team with representatives from each of three personnel 
groups is evaluating the drug abuse treatment programs* We believe 
this model allows us to maximize' the advantages and minimize the 
disadvantages of each group's Livolvement in evaluative research* 
(Author/MLP) 
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PROGRAM EVALUATION J WHO SHOULD DO IT? 



The last five years have seen an exponential growth in the evaluation 
of mental health programs. More recently drug abuse treatment programs 
have been the subject of intensive evaluation and follow-up efforts. 
Large scale outcome research efforts are being carried out by the National 
Institutes of Mental Health and the Special Action Office for Drug Abuse 
Prevention. Last month the Veterans Administration launched a full scale 
follow-up study of patients entering its drug dependent treatment centers, 
a project described in another paper on this panel. 

Concurrent with these national efforts are a number of local and regional 
efforts at evaluating drug abuse programs. A central question to evaluative 
^r>j research at all levels (and still an unanswered one for the follow-up com- 

ponent of the national VA study) is, put most simply, who should do it? That 
is, what group or agency should be responsible for the design, planning and 
execution of valid and useful evaluative studies? 

There are three principal groups that might be involved in evaluative 
research. These are: 

1. The treatment personnel themselves. Those who perform or are 
directly responsible for the therapeutic activities of the treatment program. 

2. Administrative or research personnel. Co-workers at the same 
institution, but who have no direct clinical role in the program. 

3. Outside research professionals. Little or no previous or continued 
contact with the treatment program and personnel, contracted to perform 
evaluation* 

There are several advantages and disadvantages in the involvement of 
each of these principal groups in evaluative research. I shall outline 
thesj considerations in the context of five core dimensions of evaluative 
studies. 



CO 

o 
o 



Richard N. Bale, Ph.D., is a Psychologist at the Veterans Administration 
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DecLgn Bias. Each group brings to the research design an understandably 
different bias^ Responding to their own need for survival, treatment 
personnel will tend to choose the outcome parameters sensitive to positive 
changes on the part of their patients. The more secure programs open to 
process change are likely to design evaluative research which can be used 
in a diagnostic way to improve program weaknesses. The parameters in either 
case may or may not ccincide with those criteria used by the other two 
groups . 

Non~treatment personnel may respond to their own or other outside 
interests, e.g. congressional inquiries, funding agencies, etc. These 
interests may be manifested in a set of outcome criteria that may not coincide 
with the explicit or implicit goals of the treatment personnel. This is a 
common occurrence in drug program evaluation where outside evaluators may 
choose a set of crime-related criteria while the treatment personnel are 
more concerned with psychological changes. When there exist wide discrep- 
ancies between critcrxa, programs may wittingly or unwittingly undermine 
evaluation efforts by outside personnel. 

Related to discrepancies in criteria are differences in the use made 
of research results. Treatment personnel often believe (and often rightly) 
that decisions based on research results will be selective rather than 
diagnostic. That is, 3 broad decision will be made to retain or discontinue 
a program rather than to improve it. In such cases the program of course 
has nothing to gain (other than continued existence) by participating in 
research. 

Response Bias. Roth researcher and subject may wittingly and unwittingly 
contribute to bias in responses to research instruments. The problems of 
experimenter bias have been thoroughly reviewed by Rosenthal (1) . Treatment 
personnel will tend to interpret responses to a subjective queir«:ion ("how 
are you getting along ^th your family?") in a positive direction. But it 
is also not unreasonabie to suspect that outside evaluators, who may profit 
from positive results Isx future contracts, may experience subtle pressure 
to bias their judgments. Local researchers on staff with the treatment 
personnel may experienoe similar pressure to positively bias results, in 
order to retain progranas which can be further researched. 
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Respondents may carry a wide variety of hidden agendas to the inter- 
view (or other research) situation. An "unsuccessful" ex-patient may be 
honest to treatment personnel to whom he can return for help; however, the 
same ex-patient may seek to protect the program by presenting himself to out- 
side evaluators more positively than is true. Bitter ex-patients, who may 
have been involuntarily dropped from the program, may refuse an interview 
with treatment personnel or give a falsely negative picture to any outside 
evaluator. And, unless the outside evaltiator guarantees confidentiality 
from all other parties inaluding treatment personnelr the response bias is 
potentially identical with that in an interview with treatment personnel 
themselves. 

These considerations are not limited to subjective data, because 
most follow-up studies also collect unverified, self-reported "objective" 
data on variables like employment, arrest and conviction records, drug use, 
etc. Because of bias problems in b oth directions — regardless of the 
research group — concern must be given to externally validating any self- 
reported data. 

Trust and Access. A central problem in the follow-up of drug abuse 
patients is the location of those patients who have left treatment. Such 
patients have a history of criminal and anti-social activity and many 
continue to commit crimes and use illegal drugs after they leave treatment 
programs . 

Treatment personnel may have an easier time locating such patients 
than outside researchers. They benefit from recognizability and trust 
that has been built during treatment. Ex-patients may see unfamiliar 
researchers as potential informers, or worse, disguised law officers. Any 
unfamiliar face has difficulty gaining access to the addict's door. 

A similar problem exists with any kind of process research on the 
treatment program. While outside personnel may be subject to less bias, 
they may also have limited access to the inner workings of the therapeutic 
processes, e.g. therapy groups, intimate dyadic encounters, and so forth. 

Expertise* Most treatment personnel are not trained in research 
methodology and techniques. Moreover, the location of departed ex-patients 
involves a technology which goes beyond trust. If local administrative or 
rt^search personnel are not knowledgeable, contracting outside professionals 
will be the only way to incorporate this expertise. This intervention may 



vary, however, from consultation to complete responsibility for design and 
execution. 

Experimental Control. In most clinical situations, control of rele- 
vant variables in a true experimental design is difficult. Somewhat less 
rigorous but nevertheless useful quasi-experimental designs are discussed 
by Campbell (2), in a significant article reviewing evaluative research of 
innovative programs. 

The more useful research designs involve some degree of experimental 
control, and that is likely to be more possible with treatment, administra- 
tive, or local research personnel in control. Outside evaluators rarely 
become involved in true experimental designs, e.g. randomization procedures. 
The approval of those in supervisory control is usually not sufficient for 
experimental manipulation; their active involvement and responsibility is 
essential. The most benevolent situation for experimental research is, 
of course, the program designed and planned by researchers. A formidable 
example in the drug abuse treatment field is the methadone program of Santa 
Clara County under Dr. Avram Goldstein, the source of considerable experi- 
mental research. 

A Model for the Synthesis of Personnel Groups 

At the Veterans Administration Hospital at Palo Alto, California, we 
are currently conducting an intensive evaluation of our drug abuse treatment 
effort, which includes three very different therapeutic communities and a 
methadone maintenance program.. As our eighteen month intake period is ending, 
we have randomized over 600 patients to the four treatment modalities, and 
are following up each patient for two years following his admission. Pre- 
liminary results of the acceptance of programs to randomized patients were 
presented at the Fifth National Conference on Methadone Treatment (3). 

The design, planning, and execution of this project, which is funded 
by the National Institute of Mental Health (MH-22853), represents a model 
for the synthesis of the three described personnel groups. Their involve- 
ment in this project are discussed separately below. 

1. The treatment personnel were involved in the research design and 
the selection of criteria for evaluation. Thus, outcome results were pre- 
determined to be useful to the programs themselves. Treatment personnel 



are not utilized in gathering intake data (accomplished during detoxification 
before transfer to a program), locating ex-patients, or doing follow-up 
interviews . 

2. Administrative personnel were involved in the research design, and 
were instrumental in facilitating the randomization scheme. The project is 
headed by a staff psychologist not directly attached, responsible, or super- 
visory to any of the treatment programs. 

3. Outside personnel^ Stanford University employees having no direct 
line of responsibility to the Veterans Administration, are responsible for 
the execution of the project, which is funded through Stanford. 

Three full time research assistants have offices on the detoxification 
ward, but have no clinical responsibility to that program. They meet and 
interview every patient entering the study, and are also responsible for the 
subsequent follow-up contact. The independence of the research staff and 
the complete confidentiality of the data is stated verbally and in writing 
to each patient. In this way, patient responses to the researchers cannot 
affect their treatment in any way, and are relatively free of such bias. 
Similarly, confidentiality from all parties including treatment personnel 
is guaranteed in the follow-up assessment. While this separation from 
clinical staff is explicit, the research assistants nevertheless interact 
with the clinical staff and engage in informal discussions with patients. 
This contact builds a crust and familiarity which facilitates follow-up 
efforts. 

In addition to the full time outside personnel, consultants on special 
technical problems, including cost-effectiveness accounting and multivariate 
statistical analysis are employed. 

We believe by combining the three principal personnel groups in this 
manner we have maximized the aforementioned advant-ages of each. Because 
the problems of response bias remain regardless of the responsible group, 
extensive reliability and validity studies of self-reported intake and 
follow-up data are planned. Validity checks which involve the use of public 
records for employment and legal activity, urinalysis for drug use, and 
interviews with family and friends for interpersonal data. 



In summary, the most powerful evaluative research may best be accomplished 
with the integration of various personnel groups, including treatment per- 
sonnel, local administrators and researchers, and outside professionals, in 
order to minimis ^ bias, maximize trust, expertise, and experimental ; control. 
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